A network security system typically employs techniques to detect and prevent threats and attacks from infiltrating a secured network. In some cases, signatures or patterns of known vulnerabilities and exploits are used to detect threats present in network traffic. Network traffic is matched against a set of signatures associated with known threats, and if it is determined that a portion of the traffic matches such a signature, an appropriate action is taken, such as the blocking of the affected traffic, issuance of an alert to an administrator, etc. Signatures may correspond to vulnerabilities or specific exploits targeting a vulnerability. In either case, foreknowledge of a vulnerability or exploit is needed to author a corresponding signature. For this reason, matching network traffic against a set of signatures corresponding to known threats works well for detecting future attempts to use exploits seen at least once before. However, signature matching often fails in detecting new vulnerabilities and exploits, such as variations of existing vulnerabilities and exploits.
A variation or a mutation of a known attack can cause a pattern or signature corresponding to the known attack to fail to match the variation, and yet the variation may still be an effective attack. The latest threats are frequently variants of existing threats. Several approaches are currently employed to detect threats for which a signature that would detect the threat does not currently exist, but these techniques are not always effective or efficient ways to detect variants of known threats. In one approach, vulnerability signatures are used instead of specific exploit signatures. Vulnerability signatures are more general and consequently less vulnerable to variations. However, due to their generic nature, vulnerability signatures are more expensive and difficult to implement, for example, in terms of processing requirements and resulting costs. Moreover, since it is often not possible to predict how an attacker will permute or vary existing threats, it is difficult to author vulnerability signatures that encompass all possible variants but still avoid generating excessive false positives. In another approach to detecting variants and other previously unknown threats, anomaly detection techniques—such as noting deviations from past traffic patterns for a particular host, pair of hosts, protocol, port, etc.—are used; however, such techniques consume a lot of processing resources and often result in excessive false positives and false negatives. While new signatures typically are written to cover variants and other previously unknown threats once they have been observed, such an approach leaves a protected network vulnerable to attacks until the signature set is updated, by which time the variant may already have been used to compromise a host associated with the network.
Thus, there is a need for a more effective way to detect variants of known threats.